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The  Role  of  Collateral  Information  about  Examinees 
in  Item  Parameter  Estimation 
Abstract 

Standard  procedures  for  estimating  item  parameters  in  item 
response  theory  (IRT)  ignore  collateral  information  that  may  be 
available  about  examinees,  such  as  their  standing  on  demographic 
and  educational  variables.  This  paper  describes  circumstances 
under  which  collateral  information  about  examinees  may  be  used  to 
make  inferences  about  item  parameters  more  precise,  and 
circumstances  under  which  it  must  be  used  to  obtain  correct 
inferences . 

Key  words:  Item  response  theory,  Maximum  likelihood.  Missing 

data,  Missing  information  principle. 
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Introduction 

In  typical  applications  of  item  response  theory  (IRT) , 
information  is  available  about  examinees  in  addition  to  their  item 
responses.  Familiar  examples  include  values  of  demographic 
variables  such  as  age  and  gender,  and  educational  variables  such 
as  courses  taken  and  grades  received.  Although  such  collateral 
variables  are  sometimes  employed  to  make  testing  more  efficient, 
as  when  younger  students  are  administered  easier  items  than  older 
students,  it  is  standard  practice  to  ignore  them  when  estimating 
item  parameters.  Questions  arise  as  to  whether  collateral 
information  about  examinees  can  be  exploited  during  the  course  of 
item  parameter  estimation,  and  whether  it  should  be.  This  paper 
brings  together  results  from  the  psychometric  and  statistical 
literature  to  answer  these  and  related  questions. 

The  focus  is  on  maximum  likelihood  item  parameter  estimation 
under  the  "mixed  model"  approach;  that  is,  item  parameters  are 
treated  as  fixed  effects  and  examinee  parameters  as  random 
effects.  Bock  and  Aitkin  (1981)  refer  to  the  resulting  item 
parameter  estimates  as  "marginal  maximum  likelihood"  (MML) 
estimates.  After  reviewing  MML  estimation  when  no  collateral 
information  is  present  (Case  0) ,  attention  turns  to  the  situations 
listed  below.  In  each  case,  we  contrast  the  results  of  item 
parameter  estimation  expected  when  using  the  collateral 
information  about  examinees,  with  those  expected  when  ignoring  it. 
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Case  0:  No  collateral  information. 

Under  an  IRT  model  satisfying  local  independence,  the 

probability  of  the  pattern  x  -  (x^ . x  )  of  right/wrong 

responses  to  n  test  items  from  an  examinee  with  proficiency 
parameter  9  is  given  as 

n 

f<x|*,g)  -  n  f  (x  \8,p  )  ,  (l) 

j  J  J  J 

where  each  f.  is  a  function  of  known  form  in  9  and  the  unknown, 

J 

possibly  vector-valued,  item  parameter  /?.  ,  and  f)  -  (/3^ . 0^)  . 

The  probability  of  x  for  a  randomly  selected  examinee  from  a 
population  in  which  9  has  density  g(0|a)  with  parameters  a  is  the 
expectation  of  (1) : 

h(x|£,a)  -  EQf(x|0,/S)  -  J  f(x|  9,0)  g(0|a)  d  9  . 

The  probability  of  the  data  matrix  X  -  (x^ . x^)  from  a  random 

sample  of  N  examinees  from  the  population  is  then  given  as 

P(X|£.o)  -  n  f  f(x.|  9,0)  g(9\a)  d  9  .  (2) 

i 

After  X  has  been  observed,  (2)  can  be  interpreted  as  a  likelihood 
function,  say  L0()3,ajX),  and  provides  the  basis  for  marginal 


maximum  likelihood  (MML)  estimation  of  p  and  a  (e.g. ,  Bock  and 
Aitkin,  1981;  Tsutakawa,  1984). 

If  g  belongs  to  a  known  parametric  family  with  a  finite 
number  of  parameters,  standard  maximum  likelihood  results  such  as 

A  A 

asymptotic  normality  and  consistency  in  N  hold  for  P  and  a  under 
widely  satisfied  regularity  conditions  such  as  identif iability  of 
parameters  and  continuous  first  and  second  derivatives  (Kendall 
and  Stuart,  1979). 

The  Information  Matrix 

Define  the  gradient  s(0,x),  or  s  for  short,  associated  with 
single  observation  (0 ,x)  as  the  column  vector 

s(0 ,x)  -  [s' (0 ,x) ,s' (0 ,x) ] ' 

P  -  a  - 

Jp7  ln  In  g(0|a) 

A  A 

Under  regularity  conditions,  the  MLE  (p,a)  satisfies  the 
likelihood  equation 

0  -  Z  Eq(s|x  )  =  Z  J  s(0,xL)  p(0|x.,0,a)  d0  , 

i  i 

with  p(0|x^,0,a)  -  h  ^(x^|/?,a)  f(x^|0t(0)  g(0|a)  by  Bayes  theorem. 
This  MLE  is  approximately  normal  for  large  N,  with  mean  (P,a)  and 
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covariance  given  by  the  inverse  of  the  expected  information  matrix 

N  I  ,  where 
A 

Ix  -  Ex[Ee(s|x)  Eq(s ' |x)] 

-  S  E0(s|xi)  EQ(s'|xi)  hC^l^.a)  ,  (3) 

the  summation  running  over  all  possible  response  vectors  x^ . 

Since  Ex[EQ(s |x) ]-0,  Ix  can  also  be  written  as  Varx[ EQ(s | x) ] .  The 
off-diagonal  block  of  Ix  for  0  with  a  is  zero,  so  the  asymptotic 
covariance  matrix  of  0  can  be  obtained  by  inverting  I  (>3)  - 

Ex(Ee(s^>  VSJ'5)]  "  Varx[Ee(V^- 

Recent  results  on  the  structure  of  information  matrices  in 
the  presence  of  missing  data  (Little  and  Rubin,  1987;  Louis,  1982; 
Orchard  and  Woodbury,  1972)  apply  if  (9 ,x)  is  viewed  as  the 
hypothetical  complete  datum,  x  as  the  observed  data,  and  the 
latent  variable  9  as  missing  data.  Mislevy  and  Sheehan  (1988)  use 
Orchard  and  Woodbury's  "missing  information  principle"  (MIP)  to 
show  that  an  upper  bound  for  I  is  the  expected  information 
IQV  -  EVE  (ss')  that  would  obtain  if  values  of  9  were  observed 
along  with  values  of  x: 


^x  -  ExEe(ss<) 
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-  Ex[Eq(s|x)  EQ(s ' | x) ]  +  Ex[Var0(s|x)] 

“  XX  +  I0|X  •  (4) 

I  ,  the  missing  information,  is  the  expected  variance  of  the 
“  |  X 

complete-data  gradient  vector  over  9,  given  x.  It  expresses 
variation  in  ss'  over  possible  values  of  9  that  could  have  given 
rise  to  the  observed  data  x,  averaged  over  x.  If  x  determined  9 
with  complete  accuracy,  Var  (s|x)  -  0  for  all  x,  and  no 
information  would  be  lost.  As  values  of  9  are  less  well 
determined  by  x,  however,  this  variance  increases  and  information 
about  and  a  decrease  commensurately . 

Alternative  Population  Structures 

The  preceding  sections  address  estimation  when  the  examinee 
population  can  be  expressed  in  terms  of  a  density  of  known 
parametric  form,  possibly  with  unknown  parameters  that  are 
identified.  Two  alternative  situations  are  also  relevant  to 
present  concerns:  9  distributions  whose  parameters  are  not 
identified,  and  nonparametric  estimates  of  9  distributions. 

A  parameter  is  identified  if  different  values  imply  different 
probabilities  for  X.  It  is  possible  for  some  parameters  to  be 
identified  while  others  are  not.  The  concern  in  this  presentation 
is  that  0  be  identified,  for  even  if  a  is  not,  this,  along  with 

A 

the  regularity  conditions  for  f,  insures  that  /9  enjoys  the 
asymptotic  properties  mentioned  above.  In  particular,  the  large- 
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sample  variance  of  /S  is  still  given  by  the  inverse  of  I  (/3)  .  A 
consistent  estimate  for  g  is  still  obtained  if  one  extends  the 
notion  of  consistency  to  convergence  to  the  set  of  distributions 
that  (i)  includes  the  correct  distribution  and  (ii)  imply  the  same 
probabilities  for  x  (de  Leeuw  and  Verhelst,  1986),  but  in  this 
case  asymptotic  normality  is  not  meaningful  for  the  population 
distribution. 

A 

A  consistent  nonparametric  estimate  g  of  g  can  be  obtained 

A 

along  with  if  Kiefer  and  Wolfowitz's  (1956)  regularity 
conditions  are  met.  One  useful  characterization  is  a  step 
function  with  jumps  at  a  finite  number  of  points,  where  both  the 
values  of  the  points  and  the  heights  of  the  jumps  are  estimated; 
they  correspond  to  a  in  the  parametric  solution  described  above. 
The  number  of  points  depends  upon  the  IRT  model  and  the  number  of 
test  items.  Engelen  (1987)  and  Lindsay  (1987)  have  worked  out  the 
details  for  the  1-parameter  logistic  (Rasch)  IRT  model. 

Case  1:  Collateral  information,  independent  of 
examinee  and  item  sampling. 

Suppose  that  collateral  information  y  exists  about  examinees. 
Limiting  ourselves  to  categorical  y  for  convenience,  we  can 
express  g  as  a  finite  mixture  of  K  subpopulation  densities: 

K 

g(0|a)  -I  Jk  gk($ |ak) 
k-1 


(5) 
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where  is  the  possibly  vector-valued  parameter  of  the  density  in 

subpopulation  k,  and  is  the  proportion  in  which  subpopulation  k 

is  represented  in  the  mixture.  If,  for  example,  each  is  normal 

2  2 

with  mean  and  common  variance  a  ,  then  a );  and  g  is  the 
density  of  the  mixture  of  K  normal  components,  with  a  - 

Define  as  (y^ . y^)  ,  with  y.^-1  if  examinee  i  is 

associated  with  subpopulation  k  and  0  if  not.  Suppose  a  random 
sample  (X,Y)  comprised  of  N  values  of  (x,j[)  is  drawn  from  the 
population.  Let  be  the  number  of  observations  and  the  item 
responses  from  Subpopulation  k.  In  this  situation  there  are  two 
ways  to  estimate  by  MML:  using  Y  and  ignoring  Y. 

Analysis  Using  Y 

If  the  subpopulation  structure  is  dealt  with  explicitly,  the 
likelihood  function  induced  by  (X,Y)  is 

“  n  P(^i|2I)  X  P(2i  9 •£>  de 

i 

N  K  N, 

-  n  ^  k  x  n  n“  J  P(x. \b,§)  gk(0|c*k)  de 

k  k-1  i-1  ‘ 
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-  L(?r|Y)  n  L(^,ak|Xk)  .  (6) 

k  “  " 

Noting  the  isolation  of  information  about  n  in  the  first  factor, 

A 

we  have  the  consistency  of  ir  in  N  from  standard  results  on  the 
multinomial  distribution.  The  second  factor  is  the  product  of  K 
likelihoods  with  the  same  form  as  (2),  related  by  the  presence  of 
0  in  each.  Bradley  and  Gart's  (1962)  results  on  maximum 
likelihood  in  associated  populations  apply  to  this  situation. 
Informally,  if  the  usual  ML  identifiability  and  regularity 
conditions  hold  in  each  of  a  fixed  number  of  subpopulations  for 
the  parameters  involved  in  each,  and  sample  sizes  increase  at  the 
same  rate  in  all  subpopulations,  then  standard  asymptotic 
properties  hold  for  ML  estimates  of  the  joint  parameter  set. 
Bradley  and  Gart's  theorem  and  the  results  from  Case  0  thus  imply 

A  A 

consistency  for  and  each  in  the  problem  at  hand,  with  no 

additional  regularity  conditions  beyond  those  required  by  Case  0. 

The  usual  way  of  defining  an  information  matrix  in  this 

setting  is  to  condition  on  the  observed  values  of  N,  ,  giving  N  I 

K  A  Y 

where 


ZE  [E0(s|xrjr.)  E0(s' |xi,yi)  ]  (7) 

i  l 


with 
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Ee<sl2i-Xi>  ~f  P(*l*e*Xi>  dB  • 

.  yik 

Mx^ly.)  -  /  n  [gk(S|«k)]  M 

k  u 


and,  again  by  Bayes  theorem, 

y  • 

P<*l*g,Xi>  -  h‘1<xilj[i)  fo^l 9,§)  n  (gk(tf|ak)]  ik  . 

k 

By  applying  the  MIP  repeatedly,  Mislevy  and  Sheehan  (1988) 
show  that  IeXY(^)  *  IQX(y9)  ~  IXY^  ~  IX^-^  ’  where  A5:B  means  that 
the  matrix  difference  A-B  is  at  least  positive  semidef inite . 
Additional  detail  about  the  relationships  among  expected 
information  matrices  will  be  given  under  Case  2. 

Analysis  Ignoring  y 

Following  Rubin  (1976),  we  define  "ignoring  Y"  to  mean  acting 
as  though  the  observed  item  responses  arose  from  the  marginal 
distribution  of  X,  or  /  p(X,Y)  dY.  That  is, 

Lll<£-2l£>  -  EY[Liu(£,J,a|x.y)] 

-  Ey[n  P(y.|w)  J  pUjJ*,^)  p(0|y.,Q)  d0] 
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-  n  J  p(x. |0,g)  g(9  |a)  dff 
i 

-  LQ(£,a|X)  . 

In  other  words,  ignoring  Y  means  estimating  p  without  regard  to  y- 
values  that  have  been  observed,  using  x- values  only  and  proceeding 
as  if  they  had  been  collected  under  Case  0. 

Rubin's  (1976)  theorems  on  inference  and  missing  data  can  be 
used  to  verify  whether  yields  the  correct  likelihood 
inferences  about  /3,  now  viewing  Y  as  the  missing  data  and  X  as  the 
observed  data.  A  sufficient  condition  is  that  the  missing  data  be 
missing  at  random  (MAR) :  the  probability  of  the  observed  pattern 
of  missingness  must  be  the  same  for  all  possible  values  of  the 
missing  variables.  MAR  is  satisfied  in  Case  1,  since  y-values  are 
ignored  (missing)  regardless  of  their  values.  Therefore,  Case  0 
holds  if  Y  is  ignored;  consistent  estimates  of  fi  and  g  are 
obtained;  and  the  information  matrix  is  given  by  (3). 

Case  2:  Collateral  information  used  in  examinee 
sampling  but  not  item  administration 

Suppose  that  the  examinee  population  has  the  same  mixture 
structure  given  in  (5),  but  that  examinee  sampling  is  carried  at 
rates  that  differ  from  the  jt^s .  One  possibility  is  stratified 
sampling  in  accordance  with  collateral  variables  rather  than 
random  from  the  population  as  a  whole.  In  particular,  examinees 
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★ 

are  sampled  from  subpopulation  k  at  predetermined  rates  which 
are  not  necessarily  equal  to  and  which  may  take  the  value  zero. 
One  possibility  is  convenience  sampling,  in  which  the  researcher 
may  have  little  knowledge  about  the  actual  subpopulation  sampling 
rates  his  procedures  imply.  A  practical  example  is  this:  item 
parameters  for  a  test  intended  for  all  sixth  graders  in  the  nation 
are  estimated  from  the  responses  of  students  who  happened  to  be 
present  on  a  particular  day  in  schools  that  elected  to  cooperate , 
in  only  the  states  of  Ohio  and  Indiana.  In  this  section  and  the 
next,  increasing  N  indefinitely  will  mean  increasing  all  subsample 
sizes  in  the  proportions  n  .  Again,  MML  estimation  can  be  carried 
out  either  using  or  ignoring  the  values  y  of  the  collateral 
variables  of  examinees  in  the  sample. 

Analysis  Using  Y 

When  subpopulation  membership  is  employed  under  Case  2,  the 
MML  likelihood  function  has  the  same  form  as  under  Case  1: 

*  N  K  N 

L2U(~ ”  n  \  x  n  n  /  Sk(0|Qk)  dd 

k  k-1  i-1 


-  L(**|Y)  n  L(g,ak|Xk) 
k 


(8) 
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where  L(/S,a^|X^)-l  if  ir^-O.  The  expected  information  matrix  for 
/ 3  has  the  same  form  as  in  Case  1  when  y  is  used,  namely  N 
as  given  in  ( 7 ) . 

Analysis  Ignoring  Y 

As  in  Case  1,  MAR  is  satisfied  when  Y  is  ignored.  The 
appropriate  likelihood  function  to  maximize  is  obtained  by 
marginalizing  over  the  distribution  of  6  in  accordance  with  actual 
sampling  rates : 

*  *  K  * 

g  (*l«  )  "  2  *k  gk(*|«k)  .  (9) 

k-1 


where  a  -(a^, • 


*  * 
,aK,W1 ’ ‘ • ’ ’*KJ ' 


The  likelihood  is 


L2l(g,a*|x)  -  knx  JI*  /  £(xl\6,(1)  g  (t|a  ) 


dff 


n  f  f(x.| 0,0)  g*(0|a*)  d 0  .  (10) 

i 


The  problem  reverts  again  to  Case  0,  now  with  respect  to  a 
density  g  obtained  by  reweighting  the  constituents  of  the 
original  population.  Under  the  aforementioned  regularity 
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A 

conditions,  consistency  and  asymptotic  normality  hold  for  /9  as  an 

"Jc 

estimate  of  /9,  and  consistency  holds  for  g  as  an  estimate  of  g  . 

A  consistent  estimate  of  g,  however,  may  not  be  forthcoming.  To 
reconstruct  g,  one  would  need  consistent  estimates  of  each  and 
each  .  But  it  may  be  that  some  subpopulations  have  sampling 
rates  of  zero,  so  that  their  parameters  cannot  recovered;  or  it 
may  be  that  the  effective  sampling  rates  are  not  known. 

Thus,  MML  estimation  with  samples  that  are  not  representative 
of  the  targeted  population  in  the  manner  of  Case  2  yields 
consistent  estimates  of  item  parameters  but  generally  not  of  the 
original  population.  The  estimate  of  g  is  a  computational 
byproduct  of  consistent  estimation  of  fi ,  of  little  interest  in  its 
own  right . 

The  expected  information  matrix  obtained  under  sampling  rates 
ie 

n  when  ignoring  subpopulation  membership  during  estimation  is 
N  IX(Y) ’  where 


CX(Y)  “  *  Vsl^)  E0(s'|xp  h*(5i|g,a*) 


(ID 


with  h*(x^)  -  /  f(x£\0,0)  dd .  Ix^  differs  from  Ix  only 

in  that  expectations  are  taken  with  respect  to  the  subpopulation 
weights  in  the  sampling  scheme  rather  than  their  naturally 
occurring  weights. 
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Relationships  Among  Information  Matrices 

In  analogy  to  I  (0) ,  define  by  l  (0)  the  block  of  the 
expected  information  matrix  for  item  parameters  that  would  obtain 
when  sampling  in  accordance  with  prespecified  rates  for  ^  if 
values  of  8  were  observed  as  well  as  those  of  x.  Mislevy  and 
Sheehan  (1988)  show  that  I  (0)*I  (0) ,  and,  following  Orchard 

and  Woodbury  (1972) ,  begin  to  explicate  the  relationship  among 

IeXY(^)’  XXY^’  and  XX(Y)  by  Partitioning  the  variation  of  s^ 
for  a  fixed  value  of  x: 


EYEe(V/3,,5'Z)  “  eyE0(s^|5’^)  WV1^ 

+  VarY[EQ(s^|x,jr)  ]  +  EY[Var0(s^|x,^) 


The  first  term  on  the  right  is  the  squared  average  value  of  s^ 

over  all  values  of  ^  and  9  .  The  second  is  the  variance  of  the 

average  values  of  s^  with  respect  to  8  as  £  varies.  It  represents 

variation  in  E0(s^)  explained  by  ^  beyond  that  explained  by  x. 

The  final  term  is  the  variation  in  E  (s  )  remaining  unexplained 

0  0 

after  both  x  and  £  have  been  accounted  for.  Taking  expectations 
over  x  gives 


IexY(^) 


IX(Y)(^)  +  IY|X(^)  +  I0|XY(^) 
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By  appropriately  combining  these  terms,  we  have  two  instances  of 
the  missing  information  principle,  first  when  both  9  and  y  are 
missing: 

-  Wfi>  +  [  :Y|X^>  +  *e|3nr<«>  ] 

"  XX(Y)  +  I9|X(Y)(^)  : 

then  when  only  9  is  missing: 

I9XY(^)  ”  IX(Y)(^)  +  IY|X(^)  ]  +  I9|XY<:^) 

IXY(^)  +  I0|XY^~^  • 

The  portion  of  missing  information  about  /?  that  is  recovered 

by  using  Y,  then,  is  IXY(£)  *  Xx(Y)^  “  IY|X^^'*yet  another 

instance  of  the  MIP,  with  (X,Y)  viewed  as  the  complete  data  and  X 

as  the  incomplete  data.  When  y  and  6  are  independent,  this  term 

is  zero  because  for  each  x,  E.(s0)  takes  the  same  value  at  all 

9  ~P 

values  of  y.  No  information  about  is  lost  by  ignoring  Y  in  this 
case.  When  y  and  9  are  not  independent,  the  degree  to  which 
information  about  0  increases  depends  not  simply  upon  the  strength 
of  their  relationship,  but  on  the  strength  of  their  relationship 
conditional  on  x.  There  is  less  to  be  gained  by  using  collateral 
information  when  9  is  already  well  determined  by  x  alone. 
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These  results  indicate  that  greater  benefit  accrues  from 
using  collateral  information  as  it  relates  more  strongly  to  the 
latent  variable,  and  as  less  information  is  available  from  x. 
Informal  analyses  suggest  that  in  typical  educational  and 
psychological  settings,  readily  available  collateral  variables  can 
account  for  a  third  of  the  population  variance  and  increase  the 
precision  of  f)  about  as  much  as  two  to  six  additional  test  items 
(Mislevy,  1987).  This  gain  is  substantial  in  applications  such  as 
educational  assessment  surveys,  where  an  examinee  might  receive 
only  five  items,  and  it  may  be  beneficial  in  adaptive  testing, 
where  he  may  receive  fifteen  well-chosen  items.  The  proportional 
gain  is  less  impressive  with  individual  achievement  tests,  where 
test  lengths  of  60  to  100  items  are  common. 


Example  1:  Recovering  Missing  Information 

This  example  uses  artificial  data  to  illustrate  the 
relationships  among  ,  IXY (£) »  and  Ix(Y)^-^’  and  the  source 

of  information  about  0  recoverable  by  using  collateral  variables. 
Example  3  will  give  some  indications  of  this  effect  with  actual 


data. 


Suppose  that  examinee  responses  to  two  items  follow  the  Rasch 
model;  that  is,  P(Xj|0,^.)  -  [x^  (0-/3^  >  ]  /  T  l+exp(0 -0  )  ]  .  Suppose 
further  that  and  that  the  examinee  population  consists  of 

two  equally-represented  subpopulations  with  known  distributions: 
in  Subpopulation  1,  0--1  with  probability  .5  and  0-0  with 
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probability  .5;  in  Subpopulation  2,  8— 0  or  +1,  each  with 
probability  .5.  In  the  total  population,  then,  9  takes  the  values 
(-1,0, +1)  with  probabilities  ( . 25 , . 50 , . 25) .  Table  1  shows  the 
likelihoods  of  each  of  the  four  possible  response  patterns  at  each 
of  the  three  possible  8  values.  Table  2  gives,  for  both 
subpopulations  and  the  population  as  a  whole,  expected  counts  of 
each  response  pattern  for  samples  of  1000  examinees  from  each  of 
the  two  subpopulations . 


Tables  1  and  2  about  here 


Whereas  likelihoods  in  Table  1  indicate  the  probability  of  a 
response  pattern  given  9  ,  the  posterior  probabilities  in  Table  3 
indicate  the  probability  that  an  examinee  producing  a  given 
response  pattern  has  a  particular  value  of  8.  The  posteriors  in 
the  first  panel  are  conditional  on  knowing  that  an  examinee 
belongs  to  Subpopulation  1,  or  P(0|x,y-1).  They  are  obtained 
through  Bayes  theorem,  multiplying  the  pattern's  likelihood  at 
each  8  (from  Table  1)  by  that  9's  probability  in  Subpopulation  1, 
then  normalizing.  Note  that  the  column  for  6-+1  contains  only 
zeros,  since  P(?-+l|y-l)-0.  The  second  panel  in  Table  3  gives 
posteriors  conditional  on  membership  in  Subpopulation  2,  or 
P(0|x,y-2),  and  the  third  panel  gives  the  posteriors  that  obtain 
when  subpopulation  membership  is  not  known,  or  P(0|x)  -  P(0|x,y-1) 
P(y-l)  +  P(*|x,y-2)  P(y-2). 
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Table  4  gives  the  value  of  s(5,x)  for  Item  1  at  each  item/5 
combination.  The  averages  of  these  values  within  response 
patterns  also  appear,  for  each  subpopulation  and  for  the 
population  as  a  whole.  These  averages  correspond  to 
/s(5,x)  p(5|x,y)  d 8  for  y-1  and  y-2,  and  Js(5,x)  p(0|x)  d5  ,  with 
integrations  realized  as  summations  since  8  can  take  only  three 
values .  For  example , 

ECs^lx-OO.y-l)  -  s  ^(5—  l.x-OO)  P(5— l|x-00,y-l) 

+  s  0,x-00)  P(5-  0|x-00,y-l) 

+  s^gl(5-+l,x-00)  P(5— +1 |x— 00,y— 1) 

-  . 158x. 681  +  .294X.319  +  .430x0  -  .202  . 


As  seen  in  (4),  I  is  defined  as  E  E  (ss').  The  element  for 
/3^  in  N  IqX  is  the  sum  of  squared  elements  in  the  conditional  -on- 8 
columns  of  Table  4,  with  each  square  weighted  by  (i)  the  expected 
population  count  of  the  response  pattern,  from  Table  2,  and  (ii) 
the  posterior  population  probability  of  that  8,  from  the  third 
panel  in  Table  3.  The  contribution  from  x-(0,0),  for  example,  is 
calculated  as 


N  P(x-00)  Eg[ s^1(5 , x-00) 2 ] 


-  N  P(x-00) 
+ 


[s  x( 8— l,x-00) 2  P(5— -1 jx— 00) 

s „<*-  0 ,  x-00) 2  P (5-  0 1  x-00) 
pi 
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+  sfll  (i9-+l,x-00r  P(0-+1  |x-00)  ] 
Pi  - 


553.4  x  ( . 1582x.483  +  ,2942x.452  +  ,4302x.065) 


-  34.94. 


Summing  such  values  over  all  four  response  patterns  gives  154. 00- - 
the  expected  information  about  when  observing  both  x  and  0  from 
a  total  sample  of  2000  comprised  of  1000  from  each  subpopulation. 


Tables  3  and  4  about  here 


The  element  for  (. 3^  in  jX(y)  calculated  without  knowledge 
of  examinees'  0s,  and  one  must  accordingly  average  s(0,x)  over  0 
values  within  x  before  squaring.  The  average  uses  as  weights 
p(0|x),  the  posterior  probabilities  when  y  is  not  known.  The 
contribution  from  x-(0,0)  to  N  I  .v.  is  calculated  as 


N  P(x-00)  {EQ[s^1(0,x-OO)3 }‘ 


N  P(x-00) 

[s^1(0—  l,x-00) 

P(0— l|x-00) 

+ 

S/31(5”  °’£”00) 

P (0-  0 | x-00) 

+ 

s^1(0-+l,x-OO) 

P(0— f-ljx-00) ] 

-  553.4  x  ( . 158x. 483  +  .294X.452  +  ,430x,065)2 


-  31.12. 
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Summing  such  values  over  x  gives  139.58.  The  information  about  0 
lost  when  9  is  not  observed  is  thus  154.00  -  139.58  -  14.42--about 
a  9-percent  drop. 

The  element  in  I  is  obtained  in  a  manner  similar  to  that 

Al 

for  I  except  that  since  y  is  now  used,  calculations  are 

A  ( I ; 

carried  out  separately  within  subpopulations,  then  summed.  The 
squares  being  summed  within  response  patterns  are  now  averages 
over  9  weighted  by  subpopulation  posteriors  p(0|x,y).  For 
x— ( 0 , 0 ) ,  the  contribution  to  N  is 

N  P(x-00|y-l)  P(y-l)  { E0[s  (0 ,x-00) ]y-l] )2  + 

N  P(x-00|y-2)  P(y-2)  <^[5^(0  ,x-00)  |y-2]  >2 


392.2  x  ( . 158x . 681  +  .294X.319  +  .430x0)  + 

161.2  x  (.158x0  +  .294X.775  +  ,430x.225)‘ 


-  32.89  . 

Summing  over  x  gives  146 . 26- -about  a  5 -percent  drop  from  the  upper 

limit  of  154.00  in  I  .  Thus  about  half  of  the  information  about 

“X 

0^  lost  by  not  observing  9  is  recovered  by  using  £. 

Case  3:  Collateral  information  present,  and  used  in  both 
examinee  sampling  and  item  administration 

Assume  the  same  finite  mixture  structure  as  in  Cases  1  and  2 

for  the  examinee  population  as  a  whole,  and  suppose  again  that 

* 

examinees  are  sampled  at  rates  jr,  that  may  differ  from  ?r  .  In 
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contrast  to  Cases  1  and  2,  suppose  that  examinees  are  administered 
test  items  as  a  function  of  their  £  values.  The  use  of  such 
designs  is  promoted  in  IRT,  since  for  a  given  number  of  observed 
responses,  they  can  increase  the  amount  of  information  available 
about  items  and  examinees.  In  targeted  testing,  for  example, 
pupils  in  a  lower  grade  may  be  administered  an  easier  set  of  items 
than  the  overlapping  set  administered  to  pupils  in  a  higher  grade. 
Let  be  the  rate  at  which  Item  j  is  presented  to  examinees  in 

•k-k  'k 

Subpopulation  k,  and  define  ir  k  “  ^  *jk-  F°r  convenience»  assume 

■ k"k 

that  for  each  i  there  exists  at  least  one  k  such  that  7r.,>0.  Let 

P be  the  parameters  of  the  items  that  are  presented  to 

subpopulation  k,  and  note  that  for  two  subpopulations  k  and  k' , 

B .  and  B ,,  need  not  be  disioint. 

~(k)  ~(k')  J 

Analysis  Using  Y 

If  MML  estimation  is  carried  out  using  Y,  the  likelihood 
function  for  0  and  a  is  a  pro<  ut  over  subpopulations: 


K  N, 


L3u(£,a|X,Y)  =  n  IT  J  f(x.|0,£(k))  gk(*|ak) 


de 


l  L(^(k)-ak)V 


(12) 


The  results  of  Case  0  apply  for  each  subpopulation  in  which  ^k^0 
separately,  so  by  Bradley  and  Gart's  (1962)  theorem  for  associated 
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A  A 

populations,  consistency  holds  for  those  a^s  and  for  £.  The 
information  matrix  is  N  1^,  where  1^  is  calculated  as  in  (7). 

Analysis  Ignoring  Y 

When  Y  is  ignored  under  Case  3,  it  is  not  the  only  datum  that 
is  missing;  also  missing  are  the  (hypothetical)  responses  of 
examinees  to  the  items  they  were  not  presented.  The  observed 
pattern  of  missingness  consists  therefore  of  y-values  for  all 
examinees,  and  the  identifications  of  the  items  they  were  not 
presented.  But  because  each  examinee's  y-value  was  used  to 
determine  which  items  he  or  she  would  be  presented,  the 
probability  of  this  missingness  pattern  would  have  been  different 
had  the  y-values  been  different.  MAR  does  not  hold.  Of  course 
MAR  is  merely  sufficient,  not  necessary,  for  ignorability .  Rather 
than  demonstrating  that  Rubin's  (1976)  more  complicated  necessary 
condition  for  ignorability  fails,  we  discuss  the  locus  of  the 
problem  and  present  a  counterexample. 

An  attempt  to  estimate  f)  in  Case  3  without  accounting  for 
subpopulation  membership  would  proceed  as  in  Case  2,  first 
defining  the  mixture  g  (0|at  )  as  in  (9),  then  maximizing  a 
facsimile  of  (10): 
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Equation  (13)  differs  crucially  from  (10)  in  that  for  a  given 
item,  the  distribution  g  used  in  marginalization  over  9  need  not 
be  the  correct  mixture  for  the  sample  of  examinees  administered 
that  item.  The  expectation  of  the  first  derivative  of  the 
likelihood  using  Y  (Equation  12)  with  respect  to  Item  j  is 

N  2  n  2  f  [ s  (9 ,x£)  2  *  P( 6  I  >  Yk“l)  3  de  - 

k  J  a  pj  k  J 


whereas  the  corresponding  expectation  based  on  a  likelihood 
ignoring  Y  (Equation  13)  is 

'fcfc 

N  2  *  2  /  [s  (0,xp  2  *k  p(^|xi,yk-l)]  d 9  M^ly^l)  . 

k  J  2.  pj  k 

In  general  these  gradients  are  not  proportional,  and  need  not  have 

the  same  vanishing  points.  In  particular,  the  expected  maximizing 
“A* 

values  of  L^  for  large  N  need  not  coincide  with  the  expected 
maximizing  values  of  L^,  which  are  the  true  values  of  0. 
Discrepancies  remain  as  N  increases. 

Example  2:  Inconsistent  MML  Item  Parameter  Estimates 

Let  G  be  an  equally-weighted  mixture  of  two  components:  , 

whose  mass  is  concentrated  at  6—- 1,  and  G^ ,  whose  mass  is 
concentrated  at  0-+1.  Let  y^  be  1  if  Examinee  i  belongs  to 
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Component  1  (i.e.,  has  9-- 1)  and  0  if  not;  define  y ^  analogously. 
Consider  two  test  items,  numbered  1  and  2,  to  which  examinee 
responses  follow  the  Rasch  model  with  item  parameters  /8^--l  and 
/?2— +1.  Suppose  that  Item  1  is  administered  to  a  large  sample  of 
examinees  from  Subpopulation  1  only,  and  Item  2  is  presented  to  an 
equally  large  sample  from  Subpopulation  2  only.  Consonant  with 
their  modeled  expectations,  half  the  responses  to  each  item  are 
observed  to  be  correct. 

MML  estimation  of  ,  using  £  and  taking  and  as 

known,  simplifies  to  solving  the  following  subpopulation  marginal 
probabilities  for  and  For  Item  1, 

•5  -  /  P(x1-l|^1,5)  p<0|yi-l)  d 9 

-  P(X1“l|y01, 0--1)  , 
and,  for  Item  2, 

.5  -  /  P(x2-l| 02,9)  p(fl|y2-l)  d 9 

-  P(x2-1|/32,0-+1)  , 

A  A 

whence  /9^--l  and  ^2-+l,  the  true  parameter  values. 

MML  estimation  of  ignoring  ^  but  treating  the  true  mixed 
population  distribution  G  as  known  requires,  for  j-1,2,  solving 
for  in 


Collateral  Information 


28 


-  [P(Xj-l  |/9j  ,  9—1)  x  .5]  +  [P(x.-l|/?.  ,S-+l)  x  .5]  , 

A  A 

whence  -  0.  Thus  the  incorrect  assumption  implicit  in 

ignoring  Y  under  Case  3,  that  the  examinees  taking  an  item  are  a 
random  sample  of  the  total  population,  leads  to  inconsistent  item 
parameter  estimates. 

Example  3:  1983/84  NAEP  Reading  Items 

This  section  uses  an  example  from  Mislevy  (1987)  to 
illustrates  points  from  Cases  1-3  above.  The  data  are  responses 
to  ten  items  from  the  1983/84  National  Assessment  of  Educational 
Progress  survey  of  reading  (Beaton,  1987).  The  survey  addressed 
performance  levels  in  three  subpopulations  of  students:  Age 
9/Grade  4,  Age  13/Grade  8,  and  Age  17/Grade  11.  Under  the  NAEP 
item-sampling  design,  the  first  five  items  were  administered  to 
examinees  from  just  the  Age  13  and  Age  17  samples,  while  the 
remaining  five  were  administered  to  examinees  from  all  three 
samples.  Case  3,  as  described  above,  obtains.  The  subpopulation 
sample  sizes  are  900,  1087,  and  1159. 

Mislevy  and  Bock's  (1983)  BILOG  computer  program  was  employed 
to  estimate  the  parameters  of  the  3-parameter  logistic  (3PL)  model 
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from  the  item  responses .  The  item  parameters  of  Item  j  under  the 

3PL  are  /?.-( a.,b.,c.),  and  the  form  of  the  model  is 
J  J  J  J 

P (Xj “1 | 9  , a^ , b j , c j )  “  cj  +  (l-Cj)/{l+exp[-1.7aj(0-bj)] }  , 

where  x^-1  represents  a  correct  response  and  x^-0  an  incorrect 
one.  The  b  parameter  indicates  item  difficulty;  all  other  things 
being  equal,  an  item  with  a  higher  value  of  b  is  harder  than  an 
item  with  a  lower  value. 

Aside  from  a  mild  prior  distribution  on  item  parameter 
estimates,  MML  was  used  to  estimate  item  parameters  in  two  ways, 
ignoring  and  using  subpopulation  membership.  In  the  first  run, 
g(0)  was  characterized  as  a  histogram  over  ten  prespecified 
points,  without  differentiation  by  subpopulation  membership.  This 
solution  approximates  nonparametric  estimation  of  g.  The  second 
run  used  a  modification  of  BILOG  (described  in  Mislevy,  1987)  to 
obtain  estimates  that  were  similar  in  all  ways  except  that 
examinees  were  distinguished  as  to  subpopulations,  the 
distributions  of  which  were  each  approximated  by  10-point 
histograms.  The  unit-size  and  origin  of  the  9  scale  were  set  in 
both  runs  by  standardizing  the  sample  as  a  whole. 

The  dif ferentiated-population  solution  yielded  subpopulation 
means  of  -.932,  .0708,  and  .654,  with  corresponding  standard 
deviations  of  .712,  .727,  and  .823.  Subpopulation  membership  thus 
accounts  for  roughly  40-percent  of  the  variation  among  examinees. 
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Item  parameter  estimates  from  the  two  runs  are  shown  in  Table 
5.  As  discussed  in  the  analysis  of  Case  3,  the  estimates  from  the 
second  (differentiated-population)  run  are  consistent,  while  those 
from  the  first  (undifferentiated  population)  run  need  not  be. 
Estimates  of  a  and  c  parameters  vary  little  in  the  two  solutions, 
usually  exhibiting  differences  less  than  twice  the  standard  error 
of  the  undifferentiated  solution.  The  same  is  true  for  b 
estimates  of  the  last  five  items,  which  were  administered  to  all 
examinees.  More  noticeable  differences  appear  in  the  b  estimates 
of  the  first  five  items,  which  were  administered  to  only  the  two 
older  subpopulations. 

Table  5  about  here 

The  b  estimates  from  the  differentiated  solution  exceed 
those  of  the  undifferentiated  solution  by  about  three  standard 
errors  on  the  average;  the  undifferentiated  solution  makes  these 
items  seem  too  easy.  The  differentiated  population  effectively 
discounts  rates  of  correct  response  to  these  items  relative  to  the 
other  five,  since  it  is  only  the  higher  ability  subpopulations  to 
whom  the  last  five  have  been  administered. 

Although  the  undif ferentiated-population  estimates  are 
inconsistent,  they  are  close  enough  to  the  correct  estimates  to 
make  it  meaningful  to  compare  the  relative  precisions  of  the  two 
sets.  Table  5  also  shows  the  ratios  of  squared  standard  errors; 
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they  show  average  gains  of  about  10 -percent  for  a  and  c 
parameters,  and  about  40 -percent  for  b's.  A  few  of  these  values 
are  less  than  one,  suggesting  lower  precision  in  terms  of 
estimated  variances  from  the  differentiated  population  run.  While 
expected  information  always  increases  when  collateral  information 
is  employed,  estimated  variances  can  decrease  for  two  reasons. 
First,  only  estimated  information  can  be  used  in  practice;  this  is 
a  consistent  estimate  of  expected  information,  but  it  has  a 
distribution  than  can  yield  apparent  decreases- -especially  if  the 
expected  increase  is  small.  Second,  while  expected  information 
for  a  parameter  increases,  variances  are  inverses  of  information 
matrices .  Changes  in  other  elements  in  the  information  matrix  can 
make  a  variance  larger  rather  than  smaller,  even  though  the 


corresponding  element  in  the  information  matrix  was  larger. 


Results  for  the  Rasch  Model 

The  preceding  sections  assume  the  propriety  of  examinee 
distributions  in  item  parameter  estimation,  and  discuss  merely 
whether  one  should  incorporate  collateral  information  as  well. 
This  assumption  might  appear  at  odds  with  a  primary  motivating 
goal  in  the  development  of  IRT,  namely  characterizing  test  items 
in  ways  that  do  not  depend  on  the  parameters  of  individual 
examinees  or  their  distributions.  In  one  sense,  all  IRT  models 
that  posit  distinct  parameters  for  items  and  examinees  exhibit 
this  property:  the  response  probability  for  a  given  examinee  on  a 
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given  item  depends  solely  on  the  characteristics  of  that  item  and 
that  examinee . 

Estimating  item  parameters  in  the  presence  of  examinee 
parameters  is  a  thorny  statistical  problem,  however,  because 
attempting  to  accumulate  information  about  an  item  by 
administering  it  to  an  another  examinee  introduces  a  new  unknown 
parameter.  In  an  instance  of  the  "infinitely  many  incidental 
parameters"  problem  (Neyman  and  Scott,  1948),  joint  ML  estimates 
of  the  parameters  of  a  fixed  number  of  items  are  not  consistent  in 
the  number  of  examinees  (Andersen,  1973).  MML  eliminates  examinee 
parameters  by  marginalizing  over  their  distribution,  thus 
providing  consistent  item  parameter  estimates  in  the  broad  variety 
of  situations  discussed  above.  For  most  IRT  models,  standard 
statistical  theory  offers  no  serious  alternative  to 
marginalization  to  deal  with  the  problem. 

The  exception  is  the  family  of  Rasch  models,  in  which  item 
and  examinee  parameters  are  distinct  algebraically  as  well  as 
conceptually.  The  existence  of  nontrivial  sufficienct  statistics 
for  examinee  parameters  in  Rasch  models  makes  it  possible  to 
estimate  item  parameters  by  conditional  maximum  likelihood,  or  CML 
(Andersen,  1973,  1977).  In  the  Rasch  model  for  dichotomous  items, 
for  example,  the  distribution  of  response  patterns  among  those 
with  the  same  total  score  t  depends  on  but  not  8  .  The  marginal 
probability  for  Case  0  can  be  factored  as  follows: 
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P(X|£,a)  -  n  f  f(xi| 6 ,/9)  g(0| a)  dff 
i 

-  [n  p(x  |t  ,/3)]  [n  /  p(t  \6,P)  g(0|a)  d 8} 

i  i 

-  P(X|T,£)  P(T|£,a)  .  (14) 

Now  MML  estimates  are  obtained  by  viewing  the  marginal  probability 
as  a  likelihood  after  X  has  been  observed,  and  maximizing  both 
factors  jointly  for  0  and,  if  required,  for  a  as  well.  Resulting 
estimates  of  /3  are  consistent  and  asymptotically  normal.  CML 
estimates  are  obtained  by  maximizing  only  the  first  factor  on  the 
right  in  (14).  Resulting  estimates  are  also  consistent  and 
asymptotically  normal,  but  involve  neither  assuming  nor  estimating 
anything  about  examinee  abilities  or  their  distributions. 

The  connection  between  MML  and  CML  has  been  explored  in 
recent  papers  by  Cressie  and  Holland  (1983),  Engelen  (1987), 
Kelderman  (1984),  de  Leeuw  and  Verhelst  (1986),  Lindsay  (1987), 
and  Tjur  (1982).  The  relationship  can  be  expressed  in  terms  of  a 
log- linear  model  for  counts  of  response  patterns,  the  parameters 
of  which  are  the  usual  parameters  for  items  and  parameters  a 
that  are  functions  of  the  moments  of  the  population  distribution. 


Collateral  Information 


34 

In  the  so-called  semiparametric  MML  solution,  one  obtains  ML 
estimates  for  p  along  with  a  nonparametric  ML  estimate  of  g  by 
maximizing  with  respect  to  both  p  and  a,  subject  to  constraints  on 
a  that  enforce  inequalities  that  must  hold  among  the  moments  of 
any  distribution.  One  obtains  the  CML  solution  for  p  by 
maximizing  with  respect  to  p  and  a  without  enforcing  these 
constraints.  If,  for  a  given  dataset,  the  inequalities  just 
happen  to  be  satisfied,  CML  and  semiparametric  MML  estimates  of  p 
are  numerically  identical.  Empirical  observations  suggest  that 
even  when  they  are  not  satisfied,  the  differences  between  the  two 
sets  of  item  parameter  estimates  are  virtually  indistinguishable. 
Since  the  constraints  are  satisfied  in  the  limit,  CML  and 
semiparametric  estimates  of  p  have  the  same  limiting  distribution. 
In  the  semiparametric  MML  solution,  the  second  factor  in  (14)  is 
asymptotically  uninformative. 

Assuming  a  parametric  form  for  g  makes  the  second  factor  of 
(14)  asymptotically  informative,  but  it  generally  provides  little 
information  compared  to  that  conveyed  by  the  first  factor 
(Bartholomew,  1988).  Even  the  strong  assumptions  of  normal 
distributions  for  both  0s  and  P s  leads  to  estimates  that  often 
prove  quite  serviceable  in  applied  work  (Cohen,  1979).  The  only 
way  to  get  substantially  different  MML  and  CML  estimates  of  p 
under  Case  0  is  to  make  assumptions  about  g  that  are  (i) 
capriciously  strong  and  (ii)  outrageously  inconsistent  with  the 
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data.  MML  and  CML  item  parameter  estimation  for  Rasch  item 
parameters  under  Case  0  arrive  at  the  same  destination,  even  if 
traveling  different  philosophical  paths. 

Analogous  results  for  Cases  1-3  follow  readily.  The 
following  notation  can  be  used  to  write  the  marginal  likelihood 
using  collateral  information  that  applies  in  all  cases.  Let  f) 
denote  the  parameters  of  the  items  administered  to  Examinee  i,  and 
let  p^(0)  denote  the  subpopulation  distribution  that  pertains  to 
Examinee  i.  Proceeding  as  for  (14),  we  obtain 

P<£.a|X,Y)  -  [II  P<x.|t.,g(i))]  [n  J  p(t.|0,£(i))  p.(*|a)  d&] 

-  P(X|T,g)  P(T|£,a,Y)  . 

A  number  of  interesting  implications  follow.  Note  first  that 
the  probability  again  breaks  into  a  conditional  factor- -exactly 
the  same  factor  that  appears  under  Case  0--and  a  marginal  factor, 
thus  supporting  CML  estimation. 

Second,  collateral  information  appears  only  in  the  marginal 
factor,  so  CML  estimation  proceeds  in  the  same  manner  regardless 
of  whether  collateral  information  is  available,  and,  if  it  is, 
regardless  of  which  collateral- information  case  obtains. 

Third,  the  number  of  sets  of  moment  equations  that  must  be 
satisfied  if  a  nonparametric  MML  estimate  of  each  g^  is  required 
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is  the  number  of  subpopulations  with  nonzero  sampling  rates. 

There  is  more  potential  for  slight  differences  between  CML  and  MML 
estimates  of  0  than  under  Case  0,  but  again  this  source  of 
information  vanishes  in  the  limit. 

Fourth,  while  one  can  use  MML  estimates  of  0  as  virtually 
interchangeable  with  CML  estimates  in  all  cases,  a  caveat  is  in 
order  if  item  administration  was  based  on  collateral  examinee 
information.  The  results  of  Case  3  imply  that  the  correct  MML 
estimates  are  ones  that  condition  on  the  collateral  information. 


not  used  in  either  examinee  sampling  or  item  assignment,  MML  item 
parameter  estimation  may  proceed  either  ignoring  or  using  it.  If 
.  t  is  ignored,  consistent  estimates  of  item  parameters  and  of  the 
composite  examinee  population  are  still  obtained.  If  it  is  used, 
consistent  estimates  of  item  parameters,  conditional  examinee 
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distributions ,  and  the  composite  examinee  distribution  are 
obtained.  The  expected  information  about  item  parameters  when  the 
collateral  information  is  used  equals  or  exceeds  the  expected 
information  when  it  is  ignored. 

When  collateral  information  is  used  to  sample  examinees  but 
not  to  assign  items,  MML  estimates  that  ignore  this  information 
are  consistent  for  item  parameters  but  not  for  the  composite 
examinee  population.  MML  estimates  that  do  use  the  collateral 
information  are  consistent  for  item  parameters  and  for  the 
examinee  subpopulations  that  are  included  in  the  sample,  but  not 
generally  for  the  composite  examinee  population.  Again,  expected 
information  about  parameters  when  collateral  information  is  used 
equals  or  exceeds  that  expected  when  it  is  ignored. 

When  collateral  information  is  used  to  sample  examinees  and 
to  assign  items  to  them,  MML  estimation  of  item  parameters  is  not 
generally  consistent  if  the  collateral  information  is  ignored. 

To  achieve  consistent  MML  item  parameter  estimates,  one  must  take 


collateral  variables  into  account. 
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Table  1 

Response  Pattern  Probabilities 


X 

Po.tf) 

0—1 

9-  0 

0*“+l 

0  0 

.534 

.250 

.072 

0  1 

.197 

.250 

.197 

1  0 

.197 

.250 

.197 

1  1 

.072 

.250 

.534 

Table  2 

Expected  Counts  of  Response  Patterns 
for  Subpopulation  Samples  of  1000 


X 

Subpopulation  1 

Subpopulation  2 

Total 

0  0 

392.2 

161.2 

553.4 

0  1 

223.3 

223.3 

446.6 

1  0 

223.3 

223.3 

446.6 

1  1 

161.2 

392.2 

553.4 

Total 

1000.0 

1000.0 

2000.0 
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Table  3 

Posterior  Probabilities  for  0  Given  x 


Subpopulation 

1:  P(0| 

x,y-l) 

X 

0—1 

0-  0 

0-+1 

0  0 

.681 

.319 

0 

0  1 

.440 

.560 

0 

1  0 

.440 

.560 

0 

1  1 

.225 

.775 

0 

Subpopulation  2:  P(0|x,y-2) 

X 

8—1 

o 

1 

0-+1 

0  0 

0 

.775 

.225 

0  1 

0 

.560 

.440 

1  0 

0 

.560 

.440 

1  1 

0 

.319 

.681 

Composite  population: 

P(<Mx) 

X 

8—1 

0-  0 

H 

+ 

n 

0  0 

.483 

.452 

.065 

0  1 

.220 

.559 

.220 

1  0 

.220 

.559 

.220 

1  1 

.065 

.452 

.483 
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Table  4 

Contributions  to  Gradient  for 


X 

s/31(*-2> 

VS/91(*’2)] 

0—1 

6-  0 

0“+ 1 

Subpop  1 

Subpop  2 

Composite 

0  0 

.158 

.294 

.430 

.202 

.325 

.237 

0  1 

.158 

.294 

.430 

.234 

.354 

.294 

1  0 

-  .430 

-  .294 

-.158 

-.354 

-  .234 

-  .294 

1  1 

-  .430 

-  .294 

-.158 

-.325 

-  .202 

-  .237 
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Table  5 

Item  Parameter  Estimates  and  Precision  Gains 


Run  1 : 

Ignoring  Y 

Run  2 : 

Using  Y 

Item 

A 

a 

SEj (a) 

a 

SEyCa) 

(SEy/SEj)2 

1 

.511 

.049 

.588 

.055 

.79 

2 

2.328 

.245 

2.585 

.281 

.76 

3 

1.419 

.146 

1.704 

.139 

1.10 

4 

.820 

.069 

.842 

.059 

1.37 

5 

1.432 

.131 

1.676 

.145 

.82 

6 

1.056 

.082 

1.234 

.078 

1.11 

7 

.338 

.033 

.322 

.131 

1.13 

8 

2.186 

.147 

2.939 

.134 

1.20 

9 

1.529 

.126 

1.442 

.094 

1.80 

10 

1.402 

.127 

1.605 

.118 

1.16 

Mean 

1.11 

Item 

A 

b 

SEJ(b) 

A 

b 

A 

SE^b) 

(SEU/SEI)2 

1 

-1.149 

.165 

-.842 

.132 

1.56 

2 

-  .614 

.056 

-  .383 

.049 

1.31 

3 

-.139 

.056 

.051 

.038 

2.56 

4 

.109 

.059 

.210 

.053 

1.24 

5 

-  .756 

.082 

-.482 

.062 

1.75 

6 

1.033 

.061 

1.074 

.053 

1.32 

7 

.047 

.119 

.257 

.129 

.85 

8 

1.716 

.075 

1.671 

.062 

1.46 

9 

-  .746 

.069 

-.820 

.064 

1.16 

10 

-  .361 

.055 

-  .365 

.046 

1.43 

Mean 

1.46 

Item 

A 

c 

A 

SEj (c) 

A 

C 

A 

SE^C) 

(SEy/SEj)2 

1 

.220 

.044 

.213 

.044 

1.00 

2 

.232 

.041 

.223 

.043 

.91 

3 

.238 

.029 

.240 

.027 

1.15 

4 

.143 

.027 

.120 

.026 

1.08 

5 

.193 

.046 

.205 

.044 

1.09 

6 

.180 

.013 

.205 

.012 

1.17 

7 

.161 

.026 

.203 

.025 

1.08 

8 

.126 

.008 

.123 

.007 

1.31 

9 

.237 

.037 

.208 

.038 

.95 

10 

.181 

.028 

.180 

.027 

1.08 

Mean 

1.08 
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University  of  Limburg 
P.0.  Box  616 
Maastr i cht 

The  NETHERLANDS  6200  MD 

Dr.  Timothy  Davey 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  C.  M.  Dayton 
Department  of  Measurement 
Statistics  &  Evaluation 
Co  I  I  ege  of  Education 
University  of  Maryland 
College  Park,  MD  20742 

Dr.  Ralph  J.  DeAyala 
Measurement,  Statistics, 
and  Evaluation 
Benjamin  Bldg.,  Rm.  4112 
University  of  Maryland 
College  Park,  MD  20742 

Dr.  Dattprasad  Divgi 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 

Dr.  Hei-Ki  Dong 
Bell  Communications  Research 
6  Corporate  Place 
PYA-1K226 

Piscataway,  NJ  08854 


Dr.  Fritz  Drasgow 
University  of  Illinois 
Department  of  Psychology 
603  E.  Dan i e I  St. 

Champaign,  IL  61820 

Defense  Technical 

Information  Center 
Cameron  Station,  Bldg  5 
Alexandria,  VA  22314 
Attn:  TC 
<12  Copies) 

Dr.  Stephen  Dunbar 
2248  Lindquist  Center 
for  Measurement 
University  of  Iowa 
Iowa  City,  IA  52242 

Dr.  James  A.  Earles 

Air  Force  Human  Resources  Lab 

Brooks  AFB,  TX  78235 

Dr.  Kent  Eaton 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  John  M.  Eddins 
University  of  Illinois 
252  Engineering  Research 
Laboratory 

103  South  Mathews  Street 
Urbana,  IL  61801 

Dr.  Susan  Embretson 
University  of  Kansas 
Psychology  Department 
426  Fraser 
Lawrence,  KS  66045 

Dr.  George  Englehard,  Jr. 
Division  of  Educational  Studies 
Emory  University 
210  Fishburne  Bldg. 

Atlanta,  GA  30322 

Dr.  Benjamin  A.  Fairbank 
Performance  Metrics,  Inc. 

5825  Ca I  I aghan 
Su i te  225 

San  Antonio,  TX  78228 
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I 

Or.  P-A.  Federico 

Code  51 

NPRDC 

San  Diego,  CA  92152-6800 

Dr.  Leonard  Feldt 
Lindquist  Center 
for  Measurement 
University  of  Iowa 
Iowa  City,  I A  52242 

Dr.  Richard  L.  Ferguson 
American  College  Testing 
P.0.  Box  168 
Iowa  City,  I A  52243 

Dr.  Gerhard  Fischer 
Liebiggasse  5/3 
A  1010  Vienna 
AUSTRIA 

Dr.  Myron  Fischl 
U.S.  Army  Headquarters 
OAPE-MRR 
The  Pentagon 

Washington,  DC  20310-0300 

Prof.  Donald  Fitzgerald 
University  of  New  England 
Department  of  Psychology 
Armidale,  New  South  Wales  2351 
AUSTRALIA 

Mr.  Paul  Foley 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 


Dr.  Robert  Gl aser 
Learning  Research 

&  Development  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 

Dr.  Bert  Green 
Johns  Hopkins  University 
Department  of  Psychology 
Charles  &  34th  Street 
Baltimore,  MD  21218 

DORNIER  GMBH 
P.0.  Box  1420 
D-7990  Fr i edr i chshaf en  1 
WEST  GERMANY 

Dr.  Ronald  K.  Hambleton 
University  of  Massachusetts 
Laboratory  of  Psychometric 
and  Evaluative  Research 
Hills  South,  Room  152 
Amherst,  MA  01003 

Dr.  Delwyn  Harnisch 
University  of  Illinois 
51  Gerty  Drive 
Champaign,  IL  61820 

Dr.  Grant  Henning 
Senior  Research  Scientist 
Division  of  Measurement 
Research  and  Services 
Educational  Testing  Service 
Princeton,  NJ  08541 


Dr.  Alfred  R.  Fregly 
AFOSR/NL ,  Bldg.  410 
Bolling  AFB,  DC  20332-6448 

Dr.  Robert  D.  Gibbons 
Illinois  State  Psychiatric  Inst. 
Rm  529W 

1601  W.  Taylor  Street 
Chicago,  III  60612 


Ms.  Rebecca  Hetter 
Navy  Personnel  R&D  Center 
Code  63 

San  Diego,  CA  92152-6800 

Dr.  Pau I  W  Ho  I  I  and 
Educational  Testing  Service,  21-T 
Rosedale  Road 
Princeton,  NJ  08541 


Dr.  Janice  Gifford 
University  of  Massachusetts 
School  of  Education 
Amherst,  MA  01003 


Prof.  Lutz  F.  Hornke 
Institut  fur  Psychologie 
RWTH  Aachen 
Jaegerstrasse  17/19 
D-5100  Aachen 
WEST  GERMANY 
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Or.  Paul  Horst 
677  G  Street,  #184 
Chula  Vista,  CA  92010 

Mr.  Dick  Hoshaw 
OP- 135 

Ar I i ngton  Annex 
Room  2834 

Washington,  DC  20350 

Dr.  Lloyd  Humphreys 
University  of  Illinois 
Department  of  Psychology 
603  East  Daniel  Street 
Champaign,  IL  61820 

Dr.  Steven  Hunka 
3-104  Educ.  N. 

University  of  Alberta 
Edmonton,  Alberta 
CANADA  T6G  2G5 

Dr.  Huynh  Huynh 
College  of  Education 
Univ.  of  South  Carolina 
Co  I umb i a ,  SC  29208 

Dr.  Robert  Jannarone 
Elec,  and  Computer  Eng.  Dept. 
University  of  South  Carolina 
Columbia,  SC  29208 

Dr.  Douglas  H.  Jones 
Thatcher  Jones  Associates 
P.0.  Box  6640 
10  Trafalgar  Court 
Lawrencev i I  I e ,  NJ  08648 

Dr.  Mil  ton  S.  Katz 
European  Science  Coordination 
Office 

U.S.  Army  Research  Institute 
Box  65 

FPO  New  York  09510-1500 

Prof.  John  A.  Keats 
Department  of  Psychology 
University  of  Newcastle 
N.S.W.  2308 
AUSTRALIA 


Dr.  G.  Gage  Kingsbury 

Portland  Public  Schools 

Research  and  Evaluation  Department 

501  North  Dixon  Street 

P.  0.  Box  3107 

Portland,  OR  97209-3107 

Dr .  William  Koch 
Box  7246,  Meas.  and  Eval.  Ctr. 
University  of  Texas-Austin 
Austin,  TX  78703 

Dr.  James  Kraatz 
Computer-based  Education 
Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

Dr.  Leonard  Kroeker 
Navy  Personnel  R&D  Center 
Code  62 

San  Diego,  CA  92152-6800 

Dr.  Jerry  Lehnus 

Defense  Manpower  Data  Center 

Suite  400 

1600  W i I  son  B I vd 

Rosslyn,  VA  22209 

Dr.  Thomas  Leonard 
University  of  Wisconsin 
Department  of  Statistics 
1210  West  Dayton  Street 
Madison,  WI  53705 

Dr.  Michael  Levine 
Educational  Psychology 
210  Education  Bldg. 

Un i vers i ty  of  Illinois 
Champaign,  IL  61801 

Dr.  Charles  Lewis 
Educational  Testing  Service 
Princeton,  NJ  08541-0001 

Dr.  Robert  L.  Linn 
Campus  Box  249 
University  of  Colorado 
Boulder,  CO  80309-0249 
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Or.  Robert  Lockman 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 

Dr.  Frederic  M.  Lord 
Educational  Testing  Service 
Princeton,  NJ  08541 

Or.  George  B.  Macready 
Department  of  Measurement 
Statistics  &  Evaluation 
College  of  Education 
University  of  Maryland 
College  Park,  MO  20742 

Dr.  Gary  Marco 
Stop  31-E 

Educational  Testing  Service 
Princeton,  NJ  08451 

Dr.  James  R.  McBride 
The  Psychological  Corporation 
1250  Sixth  Avenue 
San  Diego,  CA  92101 

Dr.  Clarence  C.  McCormick 
HQ,  USMEPCOM/MEPCT 
2500  Green  Bay  Road 
North  Chicago,  IL  60064 

Dr.  Robert  McKinley 
Educational  Testing  Service 
16-T 

Princeton,  NJ  08541 

Dr.  James  McMichael 
Technical  Director 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  Barbara  Means 
SRI  International 
333  Ravenswood  Avenue 
Menlo  Park,  CA  94025 

Dr.  Robert  Mi s I evy 
Educational  Testing  Service 
Princeton,  NJ  08541 


Dr.  William  Montague 

NPRDC  Code  13 

San  Diego,  CA  92152-6800 

Ms.  Kathleen  Moreno 
Navy  Personnel  R8tD  Center 
Code  62 

San  Diego,  CA  92152-6800 

Headquarters  Marine  Corps 
Code  MPI-20 
Washington,  DC  20380 

Dr.  W.  Alan  Nicewander 
University  of  Oklahoma 
Department  of  Psychology 
Norman,  OK  73071 

Deputy  Technical  Director 

NPRDC  Code  01A 

San  Diego,  CA  92152-6800 

Director,  Training  Laboratory, 
NPRDC  (Code  05) 

San  Diego,  CA  92152-6800 

Director,  Manpower  and  Personnel 
Laboratory , 

NPRDC  (Code  06) 

San  Diego,  CA  92152-6800 

Director,  Human  Factors 

&  Organizational  Systems  Lab, 
NPRDC  (Code  07) 

San  Diego,  CA  92152-6800 

Library,  NPRDC 
Code  P201L 

San  Diego,  CA  92152-6800 

Commanding  Officer, 

Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

Dr.  Harold  F.  O'Neil,  Jr. 

School  of  Education  -  WPH  801 
Department  of  Educational 
Psychology  &  Technology 
University  of  Southern  California 
Los  Angeles,  CA  90089-0031 
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Or.  James  B.  Olsen 
WICAT  Systems 
1875  South  State  Street 
Orem,  UT  84058 

Office  of  Naval  Research, 

Code  1142CS 
800  N.  Quincy  Street 
Arl ington,  VA  22217-5000 
(6  Copies) 

Office  of  Naval  Research, 

Code  125 

800  N.  Quincy  Street 
Arlington,  VA  22217-5000 

Assistant  for  MPT  Research, 
Development  and  Studies 
OP  01B7 

Washington,  DC  20370 

Dr.  Judith  Orasanu 
Basic  Research  Office 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Dr.  Jesse  Orlansky 
Institute  for  Defense  Analyses 
1801  N.  Beauregard  St. 
Alexandria,  VA  22311 

Dr.  Randolph  Park 
Army  Research  Institute 
5001  Eisenhower  Blvd. 

Alexandria,  VA  22333 

Wayne  M.  Patience 
American  Council  on  Education 
GED  Testing  Service,  Suite  20 
One  Dupont  Circle,  NW 
Washington,  DC  20036 

Dr.  James  Paulson 
Department  of  Psychology 
Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

Dept,  of  Administrative  Sciences 
Code  54 

Naval  Postgraduate  School 
Monterey,  CA  93943-5026 


Department  of  Operations  Research, 
Naval  Postgraduate  School 
Monterey,  CA  93940 

Dr.  Mark  D.  Reckase 
ACT 

P.  0.  Box  168 
Iowa  City,  I A  52243 

Dr .  Malcolm  Ree 
AFHRL/MOA 

Brooks  AFB,  TX  78235 

Dr.  Barry  Riegelhaupt 
HumRRQ 

1100  South  Washington  Street 
Alexandria,  VA  22314 

Dr.  Carl  Ross 
CNET-PDCD 
Building  90 

Great  Lakes  NTC,  IL  60088 
Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

Dr.  Fumiko  Samejima 
Department  of  Psychology 
University  of  Tennessee 
310B  Austin  Peay  Bldg. 

Knoxville,  TN  37916-0900 

Mr.  Drew  Sands 

NPRDC  Code  62 

San  Diego,  CA  92152-6800 

Lowe  I  I  Schoer 

Psychological  &  Quantitative 
Foundat i ons 
Co  I  I ege  of  Educat i on 
University  of  Iowa 
Iowa  City,  I A  52242 

Dr.  Mary  Schratz 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 

Dr.  Dan  Segal  I 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152 
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Dr.  M.  Steve  Se I Iman 
OASO  <MRA&1 ) 

2B269  The  Pentagon 
Washington,  DC  20301 

Or.  Kazuo  Shigemasu 
7-9-24  Kugenuma-Ka i gan 
Fu j i sawa  251 
JAPAN 

Dr.  William  Sims 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 

Dr.  H.  Wallace  Sinaiko 
Manpower  Research 

and  Advisory  Services 
Smithsonian  Institution 
801  North  Pitt  Street,  Suite  120 
Alexandria,  VA  22314-1713 

Or.  Richard  E.  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

Dr.  Richard  C.  Sorensen 
Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  Paul  Speckman 
University  of  Missouri 
Department  of  Statistics 
Columbia,  MO  65201 

Dr.  Judy  Spray 
ACT 

P.0.  Box  168 
Iowa  City,  IA  52243 

Dr.  Martha  Stocking 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  William  Stout 
University  of  Illinois 
Department  of  Statistics 
101  II  I  ini  Hal  I 
725  South  Wright  St. 

Champaign,  IL  61820 


Or.  Hariharan  Swaminathan 
Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01003 

Mr.  Brad  Sympson 

Navy  Personnel  R&D  Center 

Code-62 

San  Diego,  CA  92152-6800 

Dr.  John  Tangney 
AFOSR/NL ,  Bldg.  410 
Bolling  AFB,  DC  20332-6448 

Dr.  Kikumi  Tatsuoka 
CERL 

252  Engineering  Research 
Laboratory 

103  S.  Mathews  Avenue 
Urbana,  IL  61801 

Dr.  Maurice  Tatsuoka 
220  Education  Bldg 
1310  S.  Sixth  St. 

Champaign,  IL  61820 

Dr.  David  Thissen 
Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

Mr.  Gary  Thomasson 
University  of  Illinois 
Educational  Psychology 
Champaign,  IL  61820 

Dr.  Robert  Tsutakawa 
University  of  Missouri 
Department  of  Statistics 
222  Math.  Sciences  Bldg. 
Columbia,  MO  65211 

Dr.  Ledyard  Tucker 
University  of  Illinois 
Department  of  Psychology 
603  E.  Daniel  Street 
Champaign,  IL  61820 
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Or.  Vern  W.  Urry 
Personnel  R&D  Center 
Office  of  Personnel  Management 
1900  E.  Street,  NW 
Washington,  DC  20415 

Or.  David  Vale 
Assessment  Systems  Corp. 

2233  University  Avenue 

Su i te  440 

St.  Paul ,  MN  55114 

Or.  Frank  L.  Vicino 
Navv  Personnel  R&O  Center 
San  Oiego,  CA  92152-6800 

Dr.  Howard  Wainer 
Educational  Testing  Service 
Princeton,  NJ  08541 

Dr.  Ming-Mei  Wang 
Lindquist  Center 
for  Measurement 
University  of  Iowa 
Iowa  City,  I A  52242 

Dr.  Thomas  A.  Warm 
Coast  Guard  Institute 
P.  0.  Substation  18 
Oklahoma  City,  OK  73169 

Dr.  Brian  Waters 
HumRRO 

12908  Argyle  Circle 
Alexandria,  VA  22314 

Dr.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455-0344 

Dr.  Ronald  A.  We i tzman 
Box  146 

Carmel,  CA  93921 

Major  John  Welsh 

AFHRL/MOAN 

Brooks  AFB,  TX  78223 


Dr.  Douglas  Wetzel 
Code  51 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152-6800 

Dr.  Rand  R.  Wi I  cox 
University  of  Southern 
Cal i f orn i a 

Department  of  Psychology 
Los  Angeles,  CA  90089-1061 

German  Military  Representative 
ATTN:  Wolfgang  Wildqrube 
Stre i tkraef teamt 
D-5300  Bonn  2 

4000  Brandywine  Street,  NW 
Washington,  DC  20016 

Dr .  Bruce  Willi ams 
Department  of  Educational 
Psycho  I ogy 

University  of  Illinois 
Urbana,  IL  61801 

Dr.  Hilda  Wing 
NRC  MH-176 

2101  Constitution  Ave. 
Washington,  DC  20418 

Dr.  Martin  F.  Wiskoff 
Defense  Manpower  Data  Center 
550  Camino  El  Estero 
Su i te  200 

Monterey,  CA  93943-3231 

Mr.  John  H.  Wolfe 

Navy  Personnel  R&D  Center 

San  Diego,  CA  92152-6800 

Dr.  George  Wong 
Biostatistics  Laboratory 
Memorial  S I oan-Ketter i ng 
Cancer  Center 
1275  York  Avenue 
New  York,  NY  10021 

Dr.  Wallace  Wulfeck,  III 
Navy  Personnel  R&D  Center 
Code  51 

San  Diego,  CA  92152-6800 


1988/07/06 


Educational  Testing  Serv i ce/M i s I evy 


Dr.  Kentaro  Yamamoto 
03- T 

Educational  Testing  Service 
Rosedale  Road 
Princeton,  NJ  08541 

Dr.  Wendy  Yen 
CTB/McGraw  Hill 
Del  Monte  Research  Park 
Monterey,  CA  93940 

Dr.  Joseph  L.  Young 
Natio.al  Science  Foundation 
Room  320 

1800  G  Street,  N.W. 
Washington,  DC  20550 

Mr.  Anthony  R.  Zara 
National  Council  of  State 
Boards  of  Nursing,  Inc. 
625  North  Michigan  Avenue 
Suite  1544 
Chicago,  IL  60611 

Dr.  Peter  Stoloff 
Center  for  Naval  Analysis 
4401  Ford  Avenue 
P.0.  Box  16268 
Alexandria,  VA  22302-0268 


